Skip to content

fix(prose-tests): keep the walk's closing turn, and name the one live difference - #572

Merged
leeovery merged 1 commit into
mainfrom
fix/prose-closing-turn-and-known-difference
Jul 27, 2026
Merged

fix(prose-tests): keep the walk's closing turn, and name the one live difference#572
leeovery merged 1 commit into
mainfrom
fix/prose-closing-turn-and-known-difference

Conversation

@leeovery

@leeovery leeovery commented Jul 27, 2026

Copy link
Copy Markdown
Owner

The failure

Both spec-entry cases failed on a claim about a handoff block their walks emitted correctly. The block was missing from the record, so the asserter had nothing to judge and rightly refused an unevidenced step. Four walks, two cases, one cause.

It isn't the parser. Running the hook's own extraction over the settled transcript returns every turn including the missing one — so the file didn't hold that turn yet when the hook read it.

The hook runs inside the stop sequence, not after it. The agent has stopped thinking; the runtime is still writing down the last thing it said.

The fix

The turn isn't lost, just not yet on disk — the stop payload carries it as last_assistant_message, the runtime handing it over at exactly the moment it's unreadable anywhere else. It's appended as the closing turn when not already present.

Reading the transcript later from assert would also have worked, but it splits the record across two components and leaves the world holding a pointer rather than the walk. This keeps the world self-contained.

Appending-when-absent is not deduplication. Two identical turns genuinely in the transcript both survive — a step that ran twice is something the asserter must see.

Truncation, reconsidered alongside it

The caps were never about disk (the logs are throwaway) but about the asserter's prompt, which every recorded action is read into. That justifies trimming a file body no claim rests on; it doesn't justify trimming evidence.

  • walk log — caps nothing at all
  • command output — 2000 → 10000, a ceiling that bounds a runaway without ever reaching real output
  • file-read bodies — 400 unchanged, the one place content is genuinely incidental

The one known live difference, written down

An inline !`command` directive is substituted when a skill loads live. A walk reads prose as a file, so the walker takes the fallback the prose supplies for exactly that case — correct, expected, and no longer reported as a marker.

The asserter is also told what it costs: the primary arm is the one real sessions almost always take and it is never exercised here, so a claim resting on the substituted value is unprovable rather than passed. Named as the only such exemption — anything else resembling an environment quirk is a finding, not a licence.

Test plan

  • 81/81 across the prose suites, including four new cases: the closing turn arrives and in order, a turn the transcript already caught isn't double-recorded, a genuine repeat survives, and the walk log truncates nothing
  • Both spec-entry cases want a re-run to confirm they now pass on their own evidence

🤖 Generated with Claude Code

Stack

  1. docs(design): prose-tests programme design log #544
  2. feat(prose-tests): the framework — cases, worlds, runner, skill #545
  3. test(prose): feature happy-path corpus — five worlds, seven cases #546
  4. test(prose): bugfix corpus — the investigation-centric surfaces #548
  5. test: retry recursive teardown removals — kill a class of phantom failures #549
  6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
  7. docs: a contributing page for working on the system #551
  8. fix(entry-skills): every handoff arm says to invoke the skill #552
  9. fix(implementation): environment setup belongs to the setup reference alone #553
  10. fix(prose-tests): the asserter is told which substitutions were armed #554
  11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
  12. test(prose): claims assert consequences, not what was displayed #556
  13. feat(prose-tests): record everything the agents do, results included #557
  14. fix(discussion-entry): the handoff reports the source it actually had #558
  15. fix(prose-tests): the stop hook records, and names the model that walked #559
  16. fix(prose-tests): command output was never actually recorded #560
  17. feat(prose-tests): judge the walk as told, not the summary returned #561
  18. feat(prose-tests): decide in code what an agent should not be deciding #562
  19. test(prose): a case starts where a session starts #563
  20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564
  21. test(prose): give the eight read-only cases something that can fail #565
  22. test(prose): only walks that can be observed, and checks that survive the trip #566
  23. fix(prose-tests): the verdict names only the model the record names #567
  24. test(prose): discovery, walked to the point where work first exists #568
  25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
  26. docs(conventions): a step whose reference routes every exit still signposts #570
  27. test(prose): discovery's epic arm, to the same durability boundary #571
  28. fix(prose-tests): keep the walk's closing turn, and name the one live difference #572 👈 current
  29. fix(prose-tests): prescribe the cross-check, drop a check that cannot fire, lint the scope #573
  30. test(prose): declare the prose each case actually walks #574
  31. fix(investigation): the symptom interview never ran for work shaped in discovery #575
  32. feat(prose-tests): conduct — playing a person where a script cannot reach #576
  33. fix(prose-tests): what a write put in a file is evidence, not incident #577
  34. fix(investigation): the interview starts from the carrier, not over it #578
  35. test(prose): quick-fix scoping, the whole definition stage in one walk #579
  36. fix(prose-tests): the world log is the walker's record #580
  37. fix(prose-tests): a confirmed failure reports both runs' checks #581
  38. fix(prose-tests): each prescribed command runs as written #582
  39. fix(scoping): context gathering reads the whole carrier, asks only the gaps #583

This was referenced Jul 27, 2026
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 10fb586 to 2883985 Compare July 27, 2026 15:47
@leeovery
leeovery force-pushed the test/prose-discovery-epic-arm branch from a9d6186 to c976b82 Compare July 27, 2026 15:49
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 2883985 to bb9b308 Compare July 27, 2026 15:49
@leeovery
leeovery force-pushed the test/prose-discovery-epic-arm branch from c976b82 to ae932a1 Compare July 27, 2026 15:50
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from bb9b308 to d00a14a Compare July 27, 2026 15:50
@leeovery
leeovery force-pushed the test/prose-discovery-epic-arm branch from ae932a1 to d62969d Compare July 27, 2026 15:52
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from d00a14a to c15985f Compare July 27, 2026 15:52
@leeovery
leeovery force-pushed the test/prose-discovery-epic-arm branch from d62969d to 4960fba Compare July 27, 2026 15:53
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from c15985f to 5c8eaa3 Compare July 27, 2026 15:53
@leeovery
leeovery force-pushed the test/prose-discovery-epic-arm branch from 4960fba to 9e7d97c Compare July 27, 2026 15:54
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 5c8eaa3 to b25026e Compare July 27, 2026 15:54
@leeovery
leeovery force-pushed the test/prose-discovery-epic-arm branch from 9e7d97c to 96d3eac Compare July 27, 2026 15:56
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from b25026e to 72d1fbf Compare July 27, 2026 15:56
@leeovery
leeovery force-pushed the test/prose-discovery-epic-arm branch from 96d3eac to e62a792 Compare July 27, 2026 15:57
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 72d1fbf to 7f84584 Compare July 27, 2026 15:57
@leeovery
leeovery force-pushed the test/prose-discovery-epic-arm branch from e62a792 to fcba6bb Compare July 27, 2026 15:58
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 7f84584 to d5175ed Compare July 27, 2026 15:58
@leeovery
leeovery force-pushed the test/prose-discovery-epic-arm branch from fcba6bb to db544d9 Compare July 27, 2026 15:59
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from d5175ed to c992f1d Compare July 27, 2026 15:59
@leeovery
leeovery force-pushed the test/prose-discovery-epic-arm branch from db544d9 to 3766bab Compare July 27, 2026 16:00
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from c992f1d to c6d27ea Compare July 27, 2026 16:00
@leeovery
leeovery force-pushed the test/prose-discovery-epic-arm branch from 3766bab to 28f9030 Compare July 27, 2026 16:01
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from c6d27ea to ddd8f1b Compare July 27, 2026 16:01
@leeovery
leeovery force-pushed the test/prose-discovery-epic-arm branch from 28f9030 to de078e5 Compare July 27, 2026 16:02
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from ddd8f1b to 07bdc40 Compare July 27, 2026 16:02
@leeovery
leeovery changed the base branch from test/prose-discovery-epic-arm to main July 27, 2026 17:05
… difference

Both spec-entry cases failed on a claim about the handoff block their
walks emitted correctly. The block was missing from the record, so the
asserter had nothing to judge and rightly refused to pass an unevidenced
step. Four walks, two cases, one cause.

The cause is not the parser. Running the hook's own extraction over the
settled transcript returns every turn including the one it missed, so the
file simply did not hold that turn yet when the hook read it. This hook
runs inside the stop sequence rather than after it: the agent has stopped
thinking, and the runtime is still writing down the last thing it said.

The turn is not lost, only not yet on disk — the stop payload carries it
as `last_assistant_message`, which is the runtime handing it over at
exactly the moment it is unreadable anywhere else. It is appended as the
closing turn when it is not already there. Reading the transcript later
instead, from the assert command, would also have worked, but it splits
the record across two components and leaves the world holding a pointer
rather than the walk; the world stays self-contained this way.

Appending only when absent is not deduplication. Two identical turns that
are genuinely in the transcript both survive, because a step that ran
twice is something the asserter has to see.

Truncation is reconsidered with it. The caps were never about disk — the
logs are throwaway — but about the asserter's prompt, which every
recorded action is read into. That justifies trimming a file body no
claim ever rests on, and does not justify trimming evidence: the walk log
now caps nothing at all, and a command's output gets a ceiling that
bounds a runaway without ever reaching real output.

Separately, the one genuine difference between a walk and a live session
is written down rather than rediscovered each run. An inline directive is
substituted when a skill loads live; a walk reads the prose as a file, so
the walker takes the fallback the prose supplies for that case. Correct,
expected, and not a marker. The asserter is also told what it costs — the
primary arm is the one real sessions take and it is never exercised here,
so a claim resting on the substituted value is unprovable rather than
passed. It is named as the only such exemption: anything else resembling
an environment quirk is a finding.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 07bdc40 to 9fdf6ab Compare July 27, 2026 17:05
@leeovery
leeovery merged commit a59fef6 into main Jul 27, 2026
@leeovery
leeovery deleted the fix/prose-closing-turn-and-known-difference branch July 27, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant